from sklearn_benchmarks.reporting.hp_match import HpMatchReporting

scikit-learn vs. ONNX Runtime

reporting = HpMatchReporting(against_lib="onnx", config="config.yml")
reporting.make_report()

We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.

KNeighborsClassifier_brute_force

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=brute.

WARNING! Mismatch between validation scores for 12 predictions (75.0%). See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	accuracy_score_sklearn	accuracy_score_onnx	diff_accuracy_scores
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	0.663	1.000	0.337
2	KNeighborsClassifier_brute_force	predict	100000	1	100	1.000	0.757	0.243
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	0.757	0.882	0.125
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	0.882	1.000	0.118
8	KNeighborsClassifier_brute_force	predict	100000	1	100	1.000	0.757	0.243
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	0.882	0.663	0.219
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	0.757	1.000	0.243
14	KNeighborsClassifier_brute_force	predict	100000	1	100	1.000	0.922	0.078
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	0.663	0.929	0.266
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	0.896	1.000	0.104
20	KNeighborsClassifier_brute_force	predict	100000	1	2	1.000	0.922	0.078
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	0.922	0.896	0.026

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.000	6.961	0.0	-1	1	18.648	0.044	0.663	0.001	0.001	See	See
3	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.000	7.000	0.0	-1	5	0.319	0.005	1.000	0.036	0.036	See	See
6	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.000	6.851	0.0	1	100	18.714	0.074	0.882	0.001	0.001	See	See
9	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.001	6.549	0.0	-1	100	0.318	0.006	1.000	0.038	0.038	See	See
12	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.013	0.000	6.158	0.0	1	5	3.703	0.020	0.896	0.004	0.004	See	See
15	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.000	7.017	0.0	1	1	0.240	0.004	1.000	0.048	0.048	See	See
18	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.004	0.000	0.359	0.0	-1	1	3.732	0.013	0.929	0.001	0.001	See	See
21	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.004	0.000	0.367	0.0	-1	5	0.241	0.004	1.000	0.018	0.018	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.986	0.161	0.000	0.002	-1	1	0.312	0.002	1.000	6.373	6.374	See	See
2	KNeighborsClassifier_brute_force	predict	100000	1	100	0.025	0.003	0.000	0.025	-1	1	18.360	0.222	0.757	0.001	0.001	See	See
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.795	0.034	0.000	0.003	-1	5	18.269	0.000	0.882	0.153	0.153	See	See
5	KNeighborsClassifier_brute_force	predict	100000	1	100	0.026	0.003	0.000	0.026	-1	5	0.318	0.006	1.000	0.081	0.081	See	See
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.067	0.005	0.000	0.002	1	100	0.317	0.005	1.000	6.525	6.526	See	See
8	KNeighborsClassifier_brute_force	predict	100000	1	100	0.022	0.001	0.000	0.022	1	100	18.308	0.023	0.757	0.001	0.001	See	See
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.801	0.044	0.000	0.003	-1	100	18.350	0.295	0.663	0.153	0.153	See	See
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.026	0.003	0.000	0.026	-1	100	0.314	0.008	1.000	0.083	0.084	See	See
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.055	0.003	0.000	0.002	1	5	0.241	0.005	1.000	8.534	8.536	See	See
14	KNeighborsClassifier_brute_force	predict	100000	1	100	0.022	0.000	0.000	0.022	1	5	3.683	0.016	0.922	0.006	0.006	See	See
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.207	0.009	0.001	0.001	1	1	3.731	0.014	0.929	0.324	0.324	See	See
17	KNeighborsClassifier_brute_force	predict	100000	1	100	0.021	0.001	0.000	0.021	1	1	0.240	0.005	1.000	0.089	0.089	See	See
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.786	0.044	0.000	0.002	-1	1	0.239	0.004	1.000	7.467	7.468	See	See
20	KNeighborsClassifier_brute_force	predict	100000	1	2	0.009	0.005	0.000	0.009	-1	1	3.687	0.014	0.922	0.002	0.002	See	See
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.656	0.025	0.000	0.003	-1	5	3.720	0.014	0.896	0.714	0.714	See	See
23	KNeighborsClassifier_brute_force	predict	100000	1	2	0.008	0.006	0.000	0.008	-1	5	0.241	0.004	1.000	0.031	0.031	See	See

KNeighborsClassifier_kd_tree

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=kd_tree.

WARNING! Mismatch between validation scores for 12 predictions (75.0%). See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	accuracy_score_sklearn	accuracy_score_onnx	diff_accuracy_scores
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.929	1.000	0.071
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	1.000	0.946	0.054
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.946	0.951	0.005
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.951	1.000	0.049
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	1.000	0.946	0.054
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.951	0.929	0.022
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.946	1.000	0.054
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	1.000	0.911	0.089
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.929	0.894	0.035
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.891	1.000	0.109
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	1.000	0.911	0.089
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.911	0.891	0.020

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.261	0.061	0.025	0.0	-1	1	122.045	0.000	0.929	0.027	0.027	See	See
3	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.630	0.069	0.022	0.0	-1	5	2.640	0.237	1.000	1.375	1.380	See	See
6	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.622	0.059	0.022	0.0	1	100	123.718	0.000	0.951	0.029	0.029	See	See
9	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.636	0.073	0.022	0.0	-1	100	2.654	0.197	1.000	1.370	1.374	See	See
12	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.617	0.058	0.022	0.0	1	5	0.048	0.019	0.891	76.053	81.944	See	See
15	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.812	0.073	0.021	0.0	1	1	0.005	0.000	1.000	752.340	752.477	See	See
18	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.001	0.016	0.0	-1	1	0.061	0.006	0.894	0.016	0.016	See	See
21	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.000	0.030	0.0	-1	5	0.005	0.000	1.000	0.105	0.105	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.858	1.065	0.000	0.001	-1	1	2.633	0.244	1.000	0.326	0.327	See	See
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.002	0.000	0.000	0.002	-1	1	121.288	0.000	0.946	0.000	0.000	See	See
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.062	0.455	0.000	0.001	-1	5	125.518	0.000	0.951	0.008	0.008	See	See
5	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	-1	5	2.703	0.389	1.000	0.001	0.001	See	See
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	5.643	0.611	0.000	0.006	1	100	2.641	0.187	1.000	2.137	2.142	See	See
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	1	100	122.653	0.000	0.946	0.000	0.000	See	See
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	3.248	0.218	0.000	0.003	-1	100	122.767	0.000	0.929	0.026	0.026	See	See
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.005	0.001	0.000	0.005	-1	100	2.610	0.194	1.000	0.002	0.002	See	See
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.745	0.412	0.000	0.002	1	5	0.005	0.000	1.000	335.369	335.563	See	See
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.002	0.000	0.000	0.002	1	5	0.039	0.001	0.911	0.041	0.041	See	See
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.955	0.327	0.000	0.001	1	1	0.061	0.003	0.894	15.606	15.624	See	See
17	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.001	0.000	0.000	0.001	1	1	0.005	0.000	1.000	0.198	0.198	See	See
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.027	0.015	0.001	0.000	-1	1	0.005	0.000	1.000	5.255	5.257	See	See
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	1	0.038	0.001	0.911	0.062	0.062	See	See
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.023	0.001	0.001	0.000	-1	5	0.038	0.001	0.891	0.604	0.604	See	See
23	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	5	0.005	0.000	1.000	0.431	0.431	See	See

HistGradientBoostingClassifier_best

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

WARNING! Mismatch between validation scores for 1 prediction (100.0%). See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	accuracy_score_sklearn	accuracy_score_onnx	diff_accuracy_scores
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.824	1.0	0.176

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	fit	100000	100000	100	97.191	0.0	300	0.001	0.001	0.502	0.02	0.824	193.419	193.569	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.107	0.002	300	0.007	0.0	0.416	0.007	1.0	0.258	0.258	See	See